ESTI  FILE  COPY 


ESD-TR-67-69 


MTR-266 


ESD  RECORD  COPY 


RETURN  TO 

SCIENTIFIC  &  TECHNICAL  INFORMATION  DIVISION 
(ESTI),  BW10ING  121 1  . 


t 


A  COST  DATA  RESEARCH  PROGRAM  PROPOSAL  WITH  SPECIAL 
EMPHASIS  ON  A  TECHNICAL  CHARACTERISTICS  DICTIONARY 


MARCH  1967 


M.  V.  Jones 

Prepared  for 

COMPTROLLER  OFFICE 
COST  ANALYSIS  DIVISION 

ELECTRONIC  SYSTEMS  DIVISION 
AIR  FORCE  SYSTEMS  COMMAND 
UNITED  STATES  AIR  FORCE 
L.  G.  Hanscom  Field,  Bedford,  Massachusetts 


Distribution  of  this  document  is  unlimited. 


Project  1820 

Prepared  by 

THE  MITRE  CORPORATION 
Bedford,  Massachusetts 

Contract  AF19(628)-5165 


This  document  may  be  reproduced  to  satisfy  official 
needs  of  U.S.  Government  agencies.  No  other  repro¬ 
duction  authorized  except  with  permission  of  Hq. 
Electronic  Systems  Division,  ATTN:  ESTI. 


When  US  Government  drawings,  specifications,  or 
other  data  are  used  for  any  purpose  other  than  a 
definitely  related  government  procurement  operation, 
the  government  thereby  incurs  no  responsibility 
nor  any  obligation  whatsoever;  and  the  fact  that  the 
government  may  have  formulated,  furnished,  or  in 
any  way  supplied  the  said  drawings,  specifications, 
or  other  data  is  not  to  be  regarded  by  implication 
or  otherwise,  as  in  any  manner  licensing  the  holder 
or  any  other  person  or  corporation,  or  conveying 
any  rights  or  permission  to  manufacture,  use,  or  seii 
any  patented  invention  that  may  in  any  way  be  related 
thereto. 


Do  not  return  this  copy.  Retain  or  destroy. 


ESD-TR-67-69 


MTR-266 


A  COST  DATA  RESEARCH  PROGRAM  PROPOSAL  WITH  SPECIAL 
EMPHASIS  ON  A  TECHNICAL  CHARACTERISTICS  DICTIONARY 


MARCH  1967 


M.  V.  Jones 


Prepared  for 

COMPTROLLER  OFFICE 
COST  ANALYSIS  DIVISION 

ELECTRONIC  SYSTEMS  DIVISION 
AIR  FORCE  SYSTEMS  COMMAND 
UNITED  STATES  AIR  FORCE 
L.  G.  Hanscom  Field,  Bedford,  Massachusetts 


Distribution  of  this  document  is  unlimited. 


Project  1820 

Prepared  by 

THE  MITRE  CORPORATION 
Bedford,  Massachusetts 

Contract  AF  19(628)-5 165 


I 


FOREWORD 

The  writer  is  greatly  indebted  to  the  ESD  Cost  Analysis  Division  (R.  Fuller 
and  R.  Schiazza)  and  to  the  MITRE  System  Analysis  Department  (F.  A'Hearn, 
J.  Cappelletti,  S.  Lewis,  E.  Runnells)  for  reviewing  the  preliminary  drafts  of 
this  report.  The  preliminary  draft  also  benefitted  much  from  a  review  by  the 
Project  Forecast  Cost  Panel  Membership. 


REVIEW  AND  APPROVAL 


This  technical  report  has  been  reviewed  and  is  approved. 


Supervisory  Cost  Estimator 
Cost  Analysis  Division 
Comptroller 


ii 


■  ■  T<r  'Mr  **pr  - "M  If|;  ■ 


ABSTRACT 


This  report  describes  a  comprehensive  program  of  cost 
data  research  proposed  for  improving  the  credibility  and  accuracy 
of  future  cost  estimates  for  Air  Force  Systems. 


1  *1  PWt  '  — ; '  r  i  Tj — ’  ,i 


TABLE  OF  CONTENTS 

Page 

SECTION  I  INTRODUCTION  1 

RELATION  TO  THE  AFSC  L-173  SERIES  1 

THE  FIVE  BASIC  ESTIMATING  METHODS  2 

SECTION  II  MAJOR  FEATURES  OF  A  COST  DATA 

RESEARCH  PROGRAM  6 

SECTION  III  ESTABLISHING  COST  DATA  RESEARCH 

PRIORITIES  9 

ITEM  ANALYSIS  9 

ESTIMATING  METHOD  ANALYSIS  10 

SECTION  IV  THE  TECHNICAL  CHARACTERISTICS 

DICTIONARY  11 

INTRODUCTION  11 

APPLICATIONS  OF  THE  TCD  11 

SHORT-TERM  TCD  AUGMENTATION  PROGRAM  14 
LONG-TERM  PROGRAM  OF  STATISTICAL 
RESEARCH  19 

SECTION  V  IMPLEMENTATION  22 

LIST  OF  TABLES 

Table  I  Technical  Characteristics  Dictionary  12 


v 


SECTION  I 


INTRODUCTION 

RELATION  TO  THE  AFSC  L-173  SERIES 

The  AFSC  173  Letter  Series  established  a  new  set  of  costing  procedures 
to  aid  the  Command  in  managing  the  time  and  cost  required  to  obtain  technic¬ 
ally  acceptable  weapon  and  support  systems.  AFSC  L  173-1  established 
uniform  procedures  for  deriving  credible,  accurate  cost  estimates.  AFSC  L 
173-2  provided  procedures  for  collecting  cost  and  cost-related  technical  data 
in  a  standardized,  integrated  fashion  from  industrial  contractors.  AFSC  L 
173-3  provided  procedures  for  comparing  meaningfully  different  cost, 
performance,  and  schedule  estimates  from  one  time  period  to  another. 

This  report  proposes  a  comprehensive  program  of  cost  data  research. 
This  program  would  analyze  and  restructure  the  raw  data  collected  under 
the  173-2  and  173-3  Letters  and  make  these  data  available  in  form  suitable 
for  applying  the  estimating  methods  prescribed  in  173-1  to  derive  the  future 
cost  estimates  for  specific  systems. 

The  proposed  program  recognizes  that  cost  estimators  use  numerous 
methods  to  make  cost  estimates  and  that  they  need  a  different  type  of  data 
for  each  method.  For  this  reason  this  program  proposes  that  data  research 
proceed  along  multiple  lines,  rather  than  in  any  single  direction,  so  that  the 
data  base  for  all  estimating  methods  will  be  improved. 
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THE  FIVE  BASIC  ESTIMATING  METHODS 


Certain  basic  similarities  underlying  all  estimating  methods  provide  a 
conceptual  structure  around  which  a  comprehensive  program  of  Cost  Data 
Research  (CDR)  can  be  developed.  First,  all  estimating  methods  use  histor¬ 
ical  cost  experience,  in  one  fashion  or  another,  as  a  base  for  future  cost 
estimates.  Second,  although  historical  cost  experience  may  be  analyzed  in 
many  different  ways,  all  estimating  methods  can  be  classified  into  five  basic 
types.  These  similar  methodologies  apply  to  all  major  product  lines  or 
systems  (aircraft,  missiles,  space,  and  electronics)  and  also  to  all  types  of 
items  (equipment,  buildings,  or  personnel)  within  any  given  system  or  pro¬ 
gram.  The  five  basic  estimating  methods  are  discussed  below. 

Catalogues 

Much  useful  cost  information,  classified  by  type  of  item,  has  been 
collected  into  catalogues  or  compendiums.  For  example,  many  manufacturers 
publish  their  prices  for  standard,  off-the-shelf  equipment  in  available  cata¬ 
logues.  Electronic  data  processing  equipment  is  a  good  example.  Also,  cer¬ 
tain  military,  non-profit,  and  private  organizations  publish  catalogues  listing 
prices  of  equipments  of  many  different  manufacturers.  Aircraft  and  commu¬ 
nications  equipment  are  examples.  Similarly,  official  and  private  compen¬ 
diums  provide  standard  per  unit  planning  factors  for  non-equipment  items. 

For  example,  construction  cost  reference  books  give  the  cost  per  square 
foot  for  hundreds  of  types  of  standard  construction  work.  Thus,  when  the 
cost  estimator  can  identify  by  model  or  specification  the  item  to  be  costed, 
he  can  often  secure  the  cost  estimate  he  needs  directly  from  a  catalogue. 
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Specific  Analogy 


Frequently,  a  cost  estimator  uses  the  contract  or  estimated  cost  of  an 
item  of  some  single,  prior  system  as  a  basis  for  estimating  the  future  cost 
of  a  like  item  of  a  new  system.  In  searching  for  a  suitable  analagous  item, 
the  estimator  will  usually  compare  the  old  and  new  item  in  terms  of  their 
performance  or  design  characteristics  such  as  the  speed  or  weight  of  an 
aircraft  or  the  storage  capacity  of  a  computer.  This  method  is  particularly 
appropriate  when  the  new  system  will  use  the  identical  type  of  equipment 
that  the  old  system  used,  and  when  there  are  good  data  on  contract  cost 
actuals  for  the  old  system. 

Parametric  Estimating  Relationships 

Sometimes  a  cost  analyst  uses  parametric  equations  to  estimate  an 
item's  cost.  These  equations,  statistically  derived  from  multiple  historical 
observations,  express  an  item's  cost  as  a  function  of  one  or  several  per¬ 
formance,  design,  or  operational  characteristics  of  that  item.  For  example, 
the  cost  of  a  booster  in  a  new  system  may  be  estimated  as  a  function  of  the 
thrust  of  the  booster.  The  parametric  equation  estimating  method  differs 
from  the  specific  analogy  method  in  the  following  ways: 


Parametric  Equations 
Based  on  multiple  observations 


Specific  Analogy 


Frequently  based  on  a  single 
analogy 


The  relationship  is  quantified  by 
formal  statistical  analysis 


The  relationship  is  often  quali¬ 
tative  and  inferred  intuitively 


The  equations  are  frequently  The  analogies  are  usually 

derived  by  cost  methodologists  drawn  by  the  cost  estimator, 
working  independently  and  in  a  himself,  in  the  process  of 

different  time  period  than  the  making  a  specific  cost  estimate 

estimator  who  subsequently  uses 
them 
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"Rule  of  Thumb"  Factors 


For  many  items  that  the  estimator  must  cost  there  are  no  documented 
references,  e.  g.  ,  no  catalogue  records  and  no  parametric  equations.  This 
is  particularly  true  in  the  operating  and  development  cost  areas.  In  these 
cases  estimators  have  come  to  rely  on  rule-of-thumb  factors  that  express 
one  type  of  cost  as  a  simple  percentage  of  another  known  type  of  cost.  For 
example,  in  conceptual  studies,  costs  of  annual  equipment  spare  parts  are 
sometimes  estimated  as  a  percent  of  equipment  procurement  costs.  Histori¬ 
cally  25  to  50  percent  of  a  typical  system's  total  costs  have  been  estimated  by 
such  rule-of-thumb  factors. 

Expert  Sources 

Sometimes  a  cost  estimator  is  not  in  a  position  to  use  any  of  the  four 
methods  described  above.  Instead  the  estimator  receives  the  estimate  he 
needs  directly  from  an  organization  or  individual  whose  job  responsibility 
requires  him  to  be  informed  on  particular  types  of  costs.  For  example, 
an  AFSC  cost  estimator  may  obtain  an  estimate  of  the  training  costs  of  a 
new  system  from  the  Air  Training  Command  (ATC)  or  of  the  construction 
costs  from  the  Army  Engineer  (AE).  Of  course,  ATC  or  AE  will  probably 
use  one  or  several  of  the  first  four  methods  cited  as  a  basis  for  its  estimate. 

In  practice,  the  choice  of  estimating  methods  will  depend  upon,  at 
least,  four  major  considerations:  the  type  and  quantity  of  information  avail¬ 
able  describing  the  system  or  item  to  be  costed;  the  type  and  quantity  of  cost 
and  cost-related  technical  data  available  on  analogous  systems  and  items 
that  may  be  used  as  an  estimating  base;  the  relative  need  for  accuracy  in  the 
completed  cost  estimate  as  affected  by  the  type  of  management  decision  to  be 
made;  and  the  time  and  resources  available  to  do  the  estimating  job.  Where- 
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ever  possible,  it  is  preferable  to  combine  several  of  the  basic  estimating 
methods  in  costing  any  item,  because  rarely  are  the  available  data  for  using 
any  one  method  sufficiently  complete  or  accurate.  * 


A  more  extensive  discussion  of  criteria  for  selecting  a  particular  estimat¬ 
ing  method  may  be  found  in:  AFSC  L  173-1  (Cost  Estimating  Procedures), 

.  Chapter  6,  and  also  in  ESD-TR-65-396,  "Estimating  Methods  and  Data 
Sources  in  Costing  Military  Systems,"  M.  V.  Jones,  December  1965. 
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SECTION  II 


MAJOR  FEATURES  OF  A  COST  DATA  RESEARCH  PROGRAM 


Because  all  of  the  five  basic  estimating  methods  are  commonly  used, 
cost  data  research  should  proceed  in  each  of  the  following  areas. 

Catalogues 

The  number  of  cost  and  technical  characteristics  catalogues  prepared 
on  a  major-item  basis  should  be  substantially  increased.  Also,  the  quality 
of  the  data  reported  in  existing  catalogues  should  be  improved  and  such 
catalogues  should  be  updated  more  frequently.  Attachment  5  of  AFSC  L 
173-1  lists  approximately  50  different  catalogues  and  compendiums  useful 
in  costing  Air  Force  items.  However,  the  item  coverage  of  currently  avail¬ 
able  catalogues  is  very  uneven.  Some  items,  such  as  EDP  equipment,  are 
thoroughly  covered  in  three  or  four  different  catalogues  both  on  costs  and 
cost-related  technical  characteristics.  Other  items  are  not  covered  at  all. 
Also,  some  items,  such  as  aircraft,  are  much  more  intensively  catalogued 
on  technical  characteristics  than  on  costs. 

System  Data 

A  second  major  area  of  research  should  aim  to  establish  a  much  better 
program  for  consolidating,  indexing,  and  expeditiously  retrieving  historical 
data  on  individual  systems.  This  program  would  draw  upon  the  new  AFSC 
cost-tracking  system  and  upon  recent  changes  in  OSD  and  Air  Force  con¬ 
tractor  reporting  requirements.  At  present  the  information  summaries 
on  the  item  costs  of  individual  systems  or  programs  are  both  uneven  and 
non-standardized.  There  is  relatively  good  coverage  in  certain  areas; 


6 


NORAD  COST  FACTORS,  published  annually,  provides  item-by-item  cost 
breakouts  for  systems  coming  under  NORAD's  responsibility.  The  Defense 
Materials  Services,  a  private  publication,  provides  standard  descriptive 
summaries  on  the  major  features  of  most  current  military  systems;  however, 
this  source  provides  relatively  few  cost  data.  Actually,  except  for  the 
highly  aggregated  Force  Structure  and  Financial  Plan,  there  is  no  document 
that  ties  together  the  item-by-item  costs  for  all  Air  Force  systems  with 
meaningful  descriptions  of  the  items  to  which  these  costs  apply. 

Parametric  Equations 

A  third  area  of  research  should  aim  to  develop  new  and  better  para¬ 
metric  equations  for  both  hardware  and  non-equipment  items  and  for  develop¬ 
ment  and  operating  costs,  as  well  as  investment  costs.  Currently  available 
ERs,  mostly  on  equipment  items,  suffer  from  basic  data  problems;  observa¬ 
tions  are  too  few,  insufficiently  homogenous,  and  of  questionable  validity. 
These  ERs  should  be  updated  as  the  basic  data  supply  improves  both  quanti¬ 
tatively  and  qualitatively  under  the  AFSC  Cost  Information  System.  This  ER 
development  work  should  also  cover  non-financial  estimating  relationships 
as  well  as  financial  (i.  e.  ,  cost)  relationships  since  the  cost  estimator  often 
finds  such  non-financial  relationships  useful  in  completing  the  description  of 
the  system  he  will  cost.  Many  non-financial  relationships  already  appear  in 
such  documents  as  USAF's  Planning  Factors  (172-3).  Section  IV  under  "Long 
Term  Program  of  Statistical  Research"  of  this  paper  discusses  in  greater 
detail  this  ER  development  program. 
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Rule  of  Thumb  Factors 


A  fourth  major  area  of  data  research  should  investigate  the  large  number 
of  undocumented  rule-of-thumb  factors  used  in  cost  estimating.  Eventually, 
these  undocumented  rule-of-thumb  factors  should  be  replaced  with  firmly 
documented,  statistically  derived  estimating  equations.  Concurrent  with 
this  long-run  program  there  should  be  a  short-run  program  to  verify  or 
reject  the  present  rule-of-thumb  factors.  This  verification  program  should 
be  based  on  strategically  selected  spot-check  samples  covering  a  few  major 
systems.  AFSC's  efforts  should  be  coordinated  with  that  of  other  organiza¬ 
tions  working  in  this  area  such  as  USAF,  ATC,  AFLC,  RAND,  and  MITRE. 

Expert  Sources 

Two  steps  should  be  taken  to  make  a  recourse  to  expert  opinion  a  more 
reliable,  useful  method  of  cost  estimation.  One,  a  "Who's  Who"  type  of 
directory  of  the  military  costing  profession  should  be  compiled.  This 
directory  would  list  the  personnel  and  organizational  units  of  military,  non¬ 
profit,  and  industrial  organizations  working  in  the  costing  field  with  an 
indication  of  their  specialized  cost  estimating  knowledge,  indexed  primarily 
by  major  product  area.  The  Tri-Service  Costing  Group,  the  Project  Forecast 
Cost  Panel,  the  Joint  Study  Group  on  Resource  Allocation  Methodology,  and 
the  Cost/Effectiveness  Panel  of  the  Operations  Research  Society  of  America 
should  be  solicited  in  compiling  this  Directory.  Expert  opinion  should  also 
be  mobilized  to  augment  the  Technical  Characteristics  Dictionary;  this  use 
is  described  in  Section  IV  under  "Consensus  Statements". 
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SECTION  III 


ESTABLISHING  COST  DATA  RESEARCH  PRIORITIES 
ITEM  ANALYSIS 

To  ensure  that  time  and  personnel  are  used  most  effectively,  the  Cost 
Data  Research  Program  (CDRP)  should  be  introduced  selectively  on  an  item- 
by-item  basis.  *  This  means  that  a  careful  study  should  be  undertaken  to 
determine  which  of  the  hundreds  of  items  that  the  Air  Force  costs  should 
receive  the  first  and  major  consideration  in  a  CDRP.  Criteria  that  should 
be  considered  in  establishing  CDRP  priorities  are  listed  below. 

Magnitude  -  All  other  things  being  equal,  it  is  more  important 
to  do  CDR  on  large-cost  items  than  on  small-cost  items  since 
the  potential  pay-off  for  increasing  the  accuracy  of  estimated 
total  system  costs  is  greater  for  large-cost  items. 

Quality  of  Present  Data  Base  -  It  is  more  important,  other 
things  being  equal,  to  do  CDR  on  items  that  presently  lack 
good  cost  catalogues,  good  estimating  equations,  etc.  ,  than 
for  items  that  presently  have  a  relatively  strong  documented 
data  base.  New  advances  in  technology,  such  as  over-the- 
horizon  radar,  may  also  dictate  the  direction  of  CDR  emphasis. 

Level  of  Detail  -  Decisions  relative  to  the  Work  Breakdown 
Structure  level  at  which  CDR  is  to  be  conducted  should  conform 
to  the  Work  Breakdown  Structure  levels  at  which  most  cost 
_ estimation  is  done. _ 

*  The  term  "item"  is  defined  in  the  broad  sense  used  in  AFSC  L  173-1.  Item 
means  any  Program  Work  Breakdown  Structure  item,  and  includes  such 
non-equipment  items  as  system  testing  and  evaluation,  system  engineering/ 
management,  data,  etc.  as  well  as  mission  and  related  equipment  items. 
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ESTIMATING  METHOD  ANALYSIS 


The  next  step  is  to  decide  for  each  item  the  direction  that  CDR  should 
take,  i.  e.  ,  better  cost  catalogues,  better  parametric  equations,  a  new 
directory  of  knowledgeable  people,  etc.  Prior  experience  should  be  a 
major  criterion  in  making  priority  decisions;  it  is  important  to  find  out  what 
estimating  methods  and  what  data  sources  have  been  most  used  and  most 
useful  in  the  past.  Experienced  cost  estimators  can  provide  this  information. 
Whatever  the  criterion  used  for  selecting  the  specific  areas  of  CDR  and 
whatever  the  nature  of  initial  decisions,  the  allocation  of  CDR  effort  should 
be  an  iterative  decision.  Changing  requirements  plus  feedback  from  initial 
CDR  work  should  dictate  periodic  realignment  in  the  program. 
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SECTION  IV 


THE  TECHNICAL  CHARACTERISTICS  DICTIONARY 

INTRODUCTION 

The  Technical  Characteristics  Dictionary  (TCD)  was  established  by  the 
AFSC  L-173  Letter  Series,  and  is  exhibited  in  rudimentary  form  as  Attach¬ 
ment  4  of  L  173-1  and  as  Attachment  3  of  L  173-2.  Subsequently,  the  AFSC 
Product  Divisions  plus  several  industrial  and  non-profit  organizations  aug¬ 
mented  the  initial  L-173  TCD  compilations. 

Simply  stated,  the  TCD  is  a  matrix  which  lists  the  major  cost  and 
resource  related  performance,  design,  and  operating  characteristics  of 
Program  Work  Breakdown  Structure  items  which  should  be  considered  in 
deriving  and  analyzing  the  costs  of  these  items.  Table  I  illustrates  the 
basic  format  of  the  TCD. 

APPLICATIONS  OF  THE  TCD 

In  several  ways  the  TCD  aids  the  basic  estimating  methods  described 
in  Section  I  and  generally  serves  as  an  important  phase  of  a  CDRP  to  improve 
the  accuracy  of  AFSC  cost  estimates. 

Parametric  Equations 

The  TCD  provides  a  documented,  comprehensive,  systematically 
developed  list  of  technical  parameters  from  which  the  independent  variables 
can  be  selected  for  use  in  developing  new  cost-estimating  parametric 
equations.  Section  IV  under  "Long-Term  Program  of  Statistical  Research" 
discusses  the  steps  in  using  the  TCD  for  this  purpose. 
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Table  I 


Technical  Characteristics  Dictionary 


Major 

Subsystem 

Subsystem 

Lower  Level 

Breakout 

Characteristics 

Units  of 

Measure 

Air  Vehicle 

Altitude 

Range 

Dry  Weight 
etc. 

Feet 

Nautical 

miles 

Pounds 

Airframe 

AMPR  Weight 
On-Site 

Weight 

etc. 

Pounds 

Pounds 

Wing 

Area 

Span 

Material 

etc. 

Feet2 

Feet 

Type 

etc. 

etc. 

etc. 

Specific  Analogies 

The  TCD  makes  the  Specific  Analogy  method  of  cost  estimation  easier. 
A  historic  problem  in  using  the  Specific  Analogy  method  of  estimation  has 
been  the  difficulty  in  determining  how  much  alike  or  different  the  supposedly 
analogous  items  are;  it  is  not  always  clear  in  what  respects  a  new  and  an 
old  item  should  be  compared.  The  TCD  reduces  the  comparison  problem  in 
the  following  ways; 

(a)  The  TCD  tells  the  estimator  on  what  bases  he  should  compare  the 
new  and  old  items;  it  tells  him  what  parametric  values  he  should 
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seek.  In  so  doing  the  TCD  reduces  the  time  the  estimator  takes 
to  develop  an  item's  description  and  to  make  the  cost  estimate. 

(b)  The  TCD  can  facilitate  communication  between  the  cost  estimator 
and  those  who  subsequently  review  and  use  his  estimate.  Dis¬ 
agreements  and  misunderstandings  between  the  estimator  and 
reviewer  should  be  reduced  because  both  will  be  using  the  same 
guidelines  (the  TCD)  relative  to  the  parametric  values  suitable 
for  analogy. 

(c)  The  TCD  promotes  uniformity  in  the  estimating  methods  used  by 
different  estimators.  With  a  well-developed  TCD  available,  when 
two  estimators  are  confronted  with  similar  circumstances  there 
will  be  less  chance  that  one  estimator  will  give  major  considera¬ 
tion  to  a  radar's  range,  while  another  estimator  gives  major  con¬ 
sideration  to  tne  radar' s  operating  frequency  when,  perhaps,  both 
of  them  should  give  prime  consideration  to  the  radar' s  peak  power. 

The  TCD  also  facilitates  the  implementation  of  the  AFSC  Cost 
Information  System  (CIS)  by  providing  an  official  list  of  non-cost 
parameters  for  which  values  should  be  recorded  on  future  systems 
in  the  CIS. 

Expert  Source  Estimation 

A  cost  estimator  is  frequently  at  a  loss  as  to  how  to  approach  an 
expert;  the  estimator  may  not  know  how  best  to  describe  the  item  to  be 
costed,  particularly  in  a  Conceptual  Study  when  system  description  infor¬ 
mation  is  hard  to  obtain.  By  providing  an  authoritative ,  carefully  selected 
list  of  cost-related  parameters,  the  TCD  helps  the  estimator  describe  his 
requirement  to  the  technical  expert. 
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When  several  different  technical  experts  are  to  be  consulted,  the  TCD 
helps  ensure  that  the  cost  estimator  describes  the  item  to  be  costed  in  the 
same  manner  to  different  experts. 

The  TCD  tends  to  standardize  the  approach  of  different  estimators  to 
the  same  or  different  experts  on  a  similar  problem. 

The  standard  approach  helps  an  estimator  reconcile  any  differences 
in  opinions  that  he  receives  from  different  experts  and  to  combine  the  expert- 
source  and  specific-analogy  methods  of  estimation  in  making  a  given  cost 
estimate. 

Estimate  Review 

The  TCD,  if  augmented  as  described  in  Section  IV  under  "Long-Term 
Program  of  Statistical  Research,  "  will  facilitate  the  process  of  estimate 
review,  thus  indirectly  improve  the  credibility  and  accuracy  of  cost 
estimates.  Often  the  offices  reviewing  a  cost  estimate  on  a  large  complex 
system  have  only  a  generalized  knowledge  relative  to  the  characteristics  of 
the  specific  equipment  and  related  items  included  in  the  total  system  cost 
estimate.  In  the  past  it  has  sometimes  been  difficult  for  a  reviewing  office 
to  know  whether  the  parametric  analogies  used  by  the  originating  estimator 
were  the  appropriate  ones.  A  well  developed  TCD  can  help  the  reviewing 
office  make  reasoned,  systematized  evaluations  on  an  item-by-item  basis 
of  the  estimating  methodology  employed,  and  provides  an  official  point  of 
reference  for  any  questions  to  be  discussed  with  the  estimator  pertaining  to 
the  estimating  methodology. 

SHORT-TERM  TCD  AUGMENTATION  PROGRAM 

Section  IV  under  "Long-Term  Program  of  Statistical  Research"  out¬ 
lines  the  major  features  of  a  long-term  program  of  statistical  research 
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aimed  at  converting  the  present  qualitative  TCD  information  into  new  para¬ 
metric  ERs.  Preliminary  action  on  this  program  should  begin  immediately; 
however,  for  two  reasons  it  will  not  be  possible  to  develop  a  large  number  of 
usable  ERs  in  the  immediate  future.  First,  in  many  cases  the  development  of 
new,  better  ERs  must  await  the  collection  of  a  larger  quantity  and  a  more 
reliable  supply  of  data.  This  improvement  in  data  is  likely  to  come  slowly. 
Second,  apart  from  the  collection  of  data,  the  process  of  deriving  useful  ERs 
is  often  slow  and  painstaking.  Experience  has  shown  that  this  derivation  for 
a  single  item  can  consume  many  man-months  or  even  man-years  of  effort 
rather  than  man-hours  or  man-days. 

Since  better  —  even  slightly  better  —  data  are  needed  in  the  short  run, 
there  should  be  a  program,  concurrent  with  the  long  run  parametric  equation 
program,  to  refine  the  qualitative  information  presently  contained  in  the  TCD 
to  make  it  more  useful  to  cost  estimators  in  the  short-term  future.  This 
short-term  program  should  tap  the  "expertise"  that  generated  the  parameter 
listings  contained  in  the  present  TCD.  In  other  words,  an  authoritative,  per¬ 
sonalized,  relatively  undocumented  data  base  should  temporarily  be  established 
to  compensate  for  sparsely  documented,  quantitative  data.  In  effect,  this  proj¬ 
ect  would  resemble  the  procedure  followed  in  writing  AFSC  L  173-1.  AFSC 
L  173-1  basically  did  not  aim  at  developing  new  cost  methods;  it  aimed  at 
documenting  the  best  of  the  current  cost  estimating  practices  and  the  cur¬ 
rently  known  cost  estimating  principles  throughout  the  Systems  'Command  and 
its  supporting  contractors.  Similarly,  this  short-term  TCD  refinement  would 
consolidate  and  make  as  explicit  as  possible,  without  making  new  major 
statistical  studies,  all  that  is  known  about  the  parameters  listed  in  the  TCD. 

The  experts  who  generated  the  present  TCD,  including  its  recent  addenda, 
should  be  asked  to  assist  in  accomplishing  the  seven  actions  discussed  below. 
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Consolidation 


The  first  action  should  be  to  consolidate,  wherever  necessary,  the 
multiple  TCD  parametric  listings  generated  by  many  organizations  for  each 
item  into  a  single  listing.  In  some  cases  (e.  g.  ,  airframes),  the  TCD  list  of 
parameters  for  a  single  item  presently  exceeds  50.  Parameters  pertaining 
to  "storage  capacity"  for  EDP  equipment  provide  another  example  of  dupli¬ 
cation  and  overlap.  One  organization  submitted  the  following:  "storage 
capacity"  and  "capacity  of  peripheral  storage;"  another  organization  listed 
"core  storage  capacity,  "  "storage  type, "  and  "core  storage  buffers  (number). " 
Other  organizations  submitted  still  different  "storage"  parameters.  The 
question  to  be  resolved  is:  Should  there  be  two,  three,  five,  ten,  etc. 
discrete  parameters  for  EDP  storage?  This  same  type  of  question  applies 
across  the  whole  list  of  TCD  submissions  generated  to  date.  It  means  that 
a  substantial  TCD  consolidation  task  must  be  accomplished.  This  consoli¬ 
dation  will  frequently  be  tedious  because  the  different  military  and  private 
organizations  that  provided  AFSC  with  TCD  listings  sometimes  used  dif¬ 
ferent  nomenclatures  to  identify  either  a  given  item  or  parameter. 

Definition 

A  second  step  should  be  to  derive  standard  explicit  definitions  for 
each  item  and  each  parameter  listed  in  the  TCD.  None  of  the  TCD  submis¬ 
sions  that  AFSC  received  to  date  provided  definitions  for  either  the  items 
or  parameters  listed.  The  AFSC  L-173  Manuals  generally  provide  defini¬ 
tions  only  for  Work  Breakdown  Structure  Level  4  items,  which  is  one  level 
higher  than  that  at  which  most  cost  estimation  is  done,  and  one  level  higher 
than  most  TCD  listings. 
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Expansion 


The  list  of  items  presently  covered  by  the  TCD  should  be  expanded.  At 
the  present  time  the  TCD  mainly  covers  selected  equipment  and  equipment- 
related  items.  The  TCD  omits  construction,  operations  and  maintenance, 
military  personnel,  RDTE,  and  other  items  constituting  roughly  70  percent 
of  a  typical  system's  total  activity,  life-cycle  costs.  Moreover,  even  within 
the  equipment  group,  only  a  small  percentage  of  all  Work  Breakdown  Structure 
items  are  covered  by  the  TCD.  Presently,  the  TCD  provides  cost-related 
parameters  for  only  slightly  more  than  50  of  the  approximately  1000  Work 
Breakdown  Structure  Levels  4,  5,  and  6  items.  In  other  words,  the  present 
TCD  barely  scratches  the  surface  in  the  scope-of-items  coverage  required. 

Workload  Assessment 

There  should  be  an  assessment  of  the  total  workload  involved  in  raising 
the  TCD  to  its  ultimate  configuration.  A  gross  item  count,  as  suggested 
above,  is  only  part  of  the  workload  story  because  the  search  for  cost-related 
parameters  for  many  items  can  be  accomplished  concurrently.  In  this  con¬ 
nection,  coordinated  research  among  the  AFSC  Product  Divisions  can  save 
much  duplicate  effort  since  all  Product  Divisions  have  Data,  System  Engineer¬ 
ing/Management,  etc.  costs.  Even  within  a  given  Product  Division,  a  single 
set  of  parameters  might  satisfy  several  item  groups.  For  example,  similar 
parameters  might  cover  both  Operational  and  Maintenance  Trainers  in  the 
Command  and  Control  area. 

Priority 

Equal  time  and  effort  for  TCD  augmentation  should  not  be  given  to  all 
Work  Breakdown  Structure  items  and  parameters.  The  criteria  suggested 
for  establishing  priorities  for  the  total  CDRP  should  also  be  considered  in 
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establishing  item  priorities  for  TCD  research.  New  TCD  research  should 
aim  to  develop  parametric  data  for  some  important  items  not  now  included 
in  the  TCD.  Moreover,  even  for  the  items  now  covered  by  the  TCD  (equip¬ 
ment  items),  the  cost-related  characteristics  listed  by  the  TCD  have  been 
primarily  performance  and  design  parameters.  Future  TCD  work  should 
seek  to  identify  the  important  cost-related  operating  and  program  charac¬ 
teristics  such  as  activity  rates,  manning  concepts,  and  logistics  support 
concepts. 


The  present  TCD  list  of  parameters  should  be  screened  to  separate 
the  important  cost-related  characteristics  from  less  important  ones.  A 
shorter  parameter  list  is  preferred  because  it  concentrates  the  cost  esti¬ 
mator's  valuable  time  in  analyzing  the  important  parameters.  From  the 
long-run  objective  of  generating  new  formal,  statistical  relationships, 
screening  is  also  necessary  both  to  reduce  the  statistical  calculation  workload 
and  to  eliminate  parameters  not  susceptible  to  quantitative  analysis.  Also, 
parameters  should  be  deleted  from  the  TCD  lists  whose  values  are  not  likely 
to  be  known  in  the  early  phase  of  a  system's  development.  For  example,  the 
"number  of  component  parts  required"  is  not  a  very  useful  parameter  for 
future  cost  estimating  purposes.  A  parts  count  is  unlikely  to  be  known  until 
a  particular  manufacturer's  model  of  a  required  equipment  item  has  been 
specified.  However,  when  the  equipment  model  has  been  specified,  a  cata¬ 
logue  or  a  prior  contract  price  is  likely  to  be  available  as  a  cost  estimating 
base,  and  the  parts  count  information  is  likely  to  be  of  little  additional  vlaue. 

Consensus  Statements 

Wherever  possible,  a  panel  of  experts  on  an  item  should  be  designated 
to  write  a  statement  briefly  describing  the  group's  consensus  relative  to  the 
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relationship  presumed  to  exist  between  the  major  parameters  and  the  item's 
cost.  Such  panels  should  be  composed  of  engineers  having  a  good  feel  for 
future  technological  developments  in  the  specified  field,  cost  personnel  who 
have  estimated  the  costs  for  numerous  similar  items  in  prior  systems,  and 
cost  methodologists  who  have  experimented  at  developing  ERs  for  the  particu¬ 
lar  item  in  the  past.  If  carefully  drawn,  these  statements,  distilling  and 
pooling  the  insights  of  those  best  in  a  position  to  know,  will  provide  useful 
guidance  to  the  cost  estimator  until  formal  statistical  analysis  based  on 
extensive  data  can  be  completed. 

An  illustrative,  expert-consensus  statement  might  read: 

"CONSOLE  TYPE  DATA  DISPLAY  EQUIPMENT  —  Single  charac¬ 
ter  display  rate  (characters  per  second),  brightness  (foot-lamberts),  and 
resolution  requirements  (lines  per  inch)  are  the  major  parameters  affecting 
the  procurement  cost  of  console  type  display  equipment.  Past  experience 
seems  to  indicate  that,  for  units  having  resolution  requirements  exceeding 
1,  000  lines  per  inch,  resolution  requirements  are  roughly  twice  as  impor¬ 
tant  as  the  other  two  factors  in  affecting  the  equipment's  cost.  For  units  with 
less  than  1,  000  lines  per  inch  resolution  requirements,  the  three  parameters 
are  roughly  of  equal  importance  in  affecting  the  equipment's  cost.  "* 

LONG-TERM  PROGRAM  OF  STATISTICAL  RESEARCH 
Introduction 

The  long-term  program  for  TCD  augmentation  would  differ  from  the 
short-term  program  in  the  type  of  analysis  emphasized,  the  scope  of  items 

* 

The  above  statement  is  purely  illustrative.  Subsequent  expert  or  statistical 
analyses  may  reveal  that  some  other  parameters  (maximum  scan  rate,  power 
requirements,  etc. )  are  better  predicators  of  cost  than  those  cited. 
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covered,  and  the  time  horizon  over  which  the  work  extends.  The  short-term 
program  would  be  projected  over  a  period  of  months  or  a  year  or  two  at  the 
longest,  would  aim  to  achieve  a  modest  improvement  in  the  data  base  for  a 
large  number  of  items,  and  would  rely  heavily  on  a  consensus  of  expert 
opinions.  The  long-run  program,  on  the  other  hand,  would  extend  over  a 
period  of  three  to  five  years,  would  aim  to  achieve  a  major  improvement  in 
the  data  base  for  a  smaller  group  of  important  items,  and  would  rely  primarily 
upon  formal  statistical  analysis.  In  the  early  period  of  the  CDRP  the  short¬ 
term  and  long-term  TCD  augmentation  programs  would  proceed  concurrently. 

Major  Considerations 

Preliminary  Planning 

Many  of  the  features  of  the  short-run  program  would  also  be  essential 
to  the  long-run  program,  such  as  the  consolidation  and  screening  of  param¬ 
eters,  and  the  definition  of  items  and  parameters.  The  choice  of  items 
subjected  to  statistical  analysis  would  also  be  important.  Although  all  Work 
Breakdown  Structure  items  should  eventually  be  subjected  to  formal  statistical 
analysis,  the  priority  of  treatment  and  the  intensiveness  of  the  analysis  should 
vary  according  to  the  criteria  discussed  in  Section  HI. 

Data  Collection 

The  most  challenging  aspect  of  the  long-run  statistical  analysis  program 
would  be  the  collection  of  the  required  data.  Much  of  the  existing  historical 
data  is  of  limited  value  because  the  number  of  observations  is  too  small  and 
the  available  item  data  lack  clear  and  consistent  definition  from  one  system 
and  time  period  to  the  next.  Future  ER  development  should  be  coordinated 
closely  with  both  the  Cost  Information  Reports  (CIR)  and  the  Cost  Tracking 
(CT)  System  and,  whenever  possible,  should  seek  contractor  cost  "actuals" 
as  opposed  merely  to  cost  estimates. 
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Standardized  Statistical  Methods 


Standard  statistical  guidance  should  be  developed  to  apply  to  all  AFSC 
product  lines  and  to  all  items  within  each  product  line.  Although  this 
guidance  would  draw  upon  text-book  statistical  methods,  it  should  be  tailored 
expressly  to  fit  AFSC  ER  derivation.  The  style  of  presentation  should  be 
simple  and  brief.  In  addition  to  covering  conventional  regression  analysis, 
correlation  analysis,  and  tests  of  significance,  the  guidance  should  provide 
direction  on  handling  problems  particularly  troublesome  in  military  ER 
development,  such  as  extremely  small  and  non-homogenous  data  samples. 

Computerizing  the  Program 

The  statistical  analysis  work  leading  to  the  derivation  of  ERs  should 
be  computerized  to  reduce  the  manpower  and  time  required  to  get  results 
and  to  permit  a  continual  up-date  of  results  as  new  data  are  received  from 
the  CIR  and  CT  systems. 
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SECTION  V 


IMPLEMENTATION 

This  paper  aimed  at  discussing  only  the  substantive  issues  of  a  CDRP, 

the  what-should-be-done  issues.  The  "whether,  "  "when,  "  "how  much,  " 
and  "by  whom"  issues  are  beyond  the  scope  of  this  paper.  For  this  reason 
there  is  only  cursory  mention  of  the  implementation  or  administration  of  the 
program.  However,  several  general  statements  relative  to  implementation 
should  be  noted: 

(a)  The  extent  to  which  the  total  CDRP  could  be  implemented  would 
depend  importantly  on  how  quickly  the  AFSC  Divisions  can  increase  their 
present  cost  estimating  staff. 

(b)  It  has  been  presumed  that  the  total  CDRP  would  be  monitored  by 
AFSC  Headquarters  with  most  of  the  daily  details  of  the  program  carried  on 
by  the  Product  Divisions. 

(c)  Since  the  total  CDRP  would  be  quite  comprehensive  in  scope, 
there  would  be  advantages  in  coordinating  AFSC  research  with  that  of  other 
organizations,  official  and  private,  working  to  establish  better  cost  infor¬ 
mation  systems.  Some  of  these  other  organizations  are:  OSD,  USAF  Hq.  , 
other  Air  Force  Commands  (ATC,  AFLC,  etc.),  non-profit  organizations, 
industrial  contractors,  private  research-consulting  firms,  and  professional 
associations  such  as  the  Project  Forecast  Cost  Panel,  the  Tri-Service 
Costing  Group,  and  the  Joint  Study  Group  on  Resource  Allocation  Methodology. 

(d)  Close  liaison  with  organizations  and  individuals  responsible  for 
the  administration  and  modification  of  the  new  contractor  reporting  systems 
(CIR,  CIS,  CT,  etc.)  is  indispensable. 
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(e)  The  long-run  and  short-run  sectors  of  the  CDRP  must  be  reason¬ 
ably  balanced  and  compatible.  In  several  cases  it  has  been  noted  that  a 
common- line  of  action  will  promote  both  programs. 

(f)  Considering  the  likelihood  of  personnel  staffing  constraints  in 
the  discernible  future,  it  might  be  advantageous  to  introduce  different  phases 
of  the  CDRP  selectively  on  a  pilot-test  basis  in  different  Product  Divisions 
rather  than  to  establish  the  total  system  concurrently  in  all  Product  Divisions. 
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